GitHub Copilot AI pair programmer: Asset or Liability?

Automatic program synthesis is a long-lasting dream in software engineering. Recently, promising Deep Learning (DL) based solution, called Copilot, has been proposed by OpenAI and Microsoft as an industrial product. Although some studies evaluate the correctness of Copilot solutions report its issues, more empirical evaluations are necessary to understand how developers can benefit from it effectively. In this paper, we study capabilities two different programming tasks: (i) generating (and reproducing) correct efficient for fundamental algorithmic problems, (ii) comparing Copilot’s with those human programmers on set tasks. For former, assess performance functionality solving selected problems computer science, like sorting implementing data structures. latter, dataset human-provided used. The results show that capable providing almost all however, buggy non-reproducible. Moreover, difficulties combining multiple methods generate solution. Comparing humans, our ratio humans’ greater than suggestions, while generated require less effort be repaired. Based findings, if used expert projects, become asset since suggestions could comparable contributions terms quality. However, liability novice who may fail filter or non-optimal due lack expertise.